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is  presented  which'  systematizes  ancTquanti f  ies  both 
PRP  pilot  rating  procedures  and  BCP  engineering  calculation  procedure  measures  of 
system  performance  an  a  logarithmic  ratio  basis  of  test  aircraft  configurations 
conpared  to  a  knewn  seleefeqd  standard  aircraft  (ve^idf)  configuration.  The  logari¬ 
thmic  units  10  log  (BCPtestV^sta)  and  10  log  PRPtest/PRPstd)  used  in  this  system 
evaluation  method  are  termed  "decivals,  dV"  as  they  are  10  times  the  log  base 
10  of  the  ratio  of  the  BCP  and  PRP  values  obtained  during  tests  for  the  test  air¬ 
craft  configuration  ccrpared  to  the  chosen  standard  aircraft  configuration./^ 

The  system  evaluation  is  for  chosen  time  periods  of  selected  flight  operations 
which  are  critical  to  flight  safety,  such  as  nay  occur  during  takeoff,  or  approach 
to  landing  and  may  include  emergency  engine  fad  lure,  flight  control  or  instrument 
malfunction  conditions.  System  equations  are  presented  which  answer  the  question 
as  to  hew  good  is  the  test  configuration  in  relation  to  the  known  standard  con¬ 
figuration  during  these  same  flight  conditions.  Potential  BCP  measures  are  dis¬ 
cussed  and  their  correlation  with  PRP  pilot  ratings  obtained  during  flight  test  or 
flight  simulator  test  determines  their  retention  as  effective  system  performance 
and  evaluation  measures.  The  non-dimensional  logarithmic  nature  of  the  retained  BCP 
system  performance  descriptors  allows  their  oembination  by  logarithmic  summation 
and  their  correlation  with  the  PRP  pilot  ratings  is  determined.  The  oembination  of 
BCP  measures  having  the  highest  correlation  with  pilot  ratings  is  retained  for 
final  system  evaluation. 
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METRIC  CONVERSION  FACTORS 


temperature  subtracting  temperature 


SUMMARY 


A  complete  systems  engineering  method  is  presented  which 
provides  a  missing  frame  of  reference  and  a  procedure  including 
new  measurement  units  which  quantifies  and  simplifies  the  evalua¬ 
tion  of  piloted  aircraft  systems  tested  in  flight  or  the  systems 
evaluation  of  any  man  operated  vehicle  or  machine  tested  in  its 
operational  environment. 

New  pilot  rating  procedures,  PRP,  and  new  engineering  cal¬ 
culation  procedures,  ECP,  are  presented  comparing  the  test  air¬ 
craft  configuration  ratings  and  measures  with  a  chosen  standard 
aircraft  configuration  using  non-dimensional  logarithmic  units 
termed  decivals,  dV. 

A  system  evaluation  equation  is  presented  which  quantifies 
and  describes  the  relationship  between  the  flight  test  pilot 
ratings  and  the  measured  flight  test  data.  This  method  may  be 
used  for  evaluation  purposes  during  system  design,  test,  certi¬ 
fication  of  airworthiness  and  flight  safety  evaluations  to  deter¬ 
mine  the  aircraft  in  flight  man-machine  pilot-aircraft  (operator 
vehicle)  operating  system  relationship: 


PRP 


test 


ECP 


PRPstd 


=  ki 


test 


ECPstd 


(1) 


The  answer  to^D  how  good  the  aircraft  or  vehicle  is,  regarding 
the  operationally  defined  system  evaluation  goal,  given  an 
engineering  calculation  procedure  ECP  measure  and^  given  a 
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pilot  or  operator  rating  procedure  PRP  rating  what  should  the 
ECP  measured  value  be  is  provided  by  the  above  equation  in 
logarithmic  form  as  follows: 


The  effectiveness  of  the  completed  system  evaluation  may 
be  determined  by  the  product  moment  correlation  coefficient, 

JL  2 

>/*^PRP  .  ECP  obtained  and  showing  the  amount  of 

PRP  •  ECP 

common  variance  between  pilot  ratings  and  flight  test  data. 


2 


.4! 


INTRODUCTION 


A  system  is  defined  as  any  machine  or  tool  including  all 
modes  of  transportation  vehicles,  or  their  roadways,  aerospace- 
ways,  and  seaways,  or  signs  and  signals  used  by  man  to  attain 
some  goal.  All  systems  include  the  effects  of  the  changing 
man-made  or  natural  environment  in  which  they  are  used  and 
operated  in  real  life  and  real  time. 

There  are  therefore  a  great  many  extraordinarily  different 
kinds  of  systems,  all  of  these  systems  may  be  evaluated  to 
determine  exactly  how  useful  they  are  in  attaining  the  various 
goals  that  have  been  set  for  them.  It  can  be  seen  that  this 
depends  on  the  experience  or  ability  of  the  user  or  operator 
and  the  characteristics  of  the  machine  or  tool  or  vehicle  or 
roadway  or  sign  or  signal  and  the  changing  man-made  and  natural 
environment  in  which  it  is  used  or  operated  in  real  life  and 
real  time. 


There  is  therefore  a  great  need  for  an  objective  and  reli¬ 
able  method  of  evaluating  systems.  There  is  at  the  present 
time  no  complete  unifying  method  which  provides  a  frame  of 
reference,  and  which  defines,  quantifies  and  describes  the  re¬ 
lationship  between  the  criteria  for  evaluation  and  the  observable 


and  measureable  data  regarding  the  system 
attaining  its  set  goal. 


Today  there  are  many  ways  to  describe  and  evaluate  the 
separate  and  component  parts  of  the  system.  For  example  there 
are  mathematical  models  of  the  human  operator  behavior  which 
are  summarized  and  explained  in  Reference  1  and  in  the  numerous 
references  cited  therein. 

There  are  many  excellent  rating  scales  such  as  the 
Cooper-Harper  pilot  rating  of  aircraft  handling  qualities 
described  in  Reference  2  and  its  references  lead  the  reader 
to  other  methods  available  for  obtaining  pilot  ratings^)- 

A  summary  of  the  definition  and  measurement  of  perceptual 
and  mental  workload  is  contained  in  Reference  3  which  points 
out  that  the  common  method  of  assessing  handling  qualities 
relies  on  the  subjective  opinion  of  experienced  test  pilots . 

Reference  1  emphasizes  that  these  subjective  pilot  opinions 
have  proved  to  be  quite  reliable  but  that  they  are  sometimes 
difficult  to  use  for  design  purposes.  It  also  stresses  that 
the  ultimate  goal  in  building  systems  engineering  models  of 
operator  behavior  is  to  use  these  models  in  the  evaluation 
of  aircraft  handling  qualities  essential  for  flight  operations. 
Of  utmost  importance  is  the  acknowledgement  that  these 
handling  qualities  are  vital  for  flight  safety^1). 


4 


But  there  is  a  severe  limitation  pointed  out  in  Reference  1 


r  -**■ 


regarding  these  engineering  models  which  have  been  postulated 
to  date ,  quote : 

"A  rather  severe  limitation  of  both  engineering  models 
lies  in  the  fact  that  they  lack  a  direct  connection  with 
the  pilot's  subjective  opinion  of  the  workload  involved 
in  a  given  control  situation.  In  the  end  it  is  this 
expert  opinion,  which  decides  on  the  acceptability,  or 
otherwise,  of  an  aspect  of  the  aircraft's  handling  quali¬ 
ties.  There  exists,  as  yet,  no  generally  applicable 
method  to  derive  the  pilot's  workload  from  the  parameters 
of  the  engineering  model.  There  are  serious  doubts 
whether  such  a  method  will  ever  be  developed. 

An  explanation  of  this  very  real  shortcoming  of  most 
existing  engineering  models  is  considered  to  lie  in  the 
fact  that  such  models  portray  primarily  the  control 
aspect  of  the  human  operators  activities,  thereby  ignor¬ 
ing  or  bypassing  the  equally  important  mental  activities 
of  data  processing  and  decision  making^.  It  seems, 
therefore,  that  there  is  a  need  for  new  developments, 
explicitly  combining  into  a  single  mathematical  model 
the  internal  mental  processes  of  data  handling  and 
decision  making  going  on  in  the  human  brain,  with  the 
more  overt  control  activities  previously  discussed. 
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It  turns  out  to  be  a  truly  interdisciplinary  study 
for  which  an  accepted  frame  of  reference  has  not  yet 
been  established,  and  iteration  between  rather  general 
hypothesis,  test  of  methods,  and  detailed  analysis  is 
necessary. " 

This  report  is  intended  to  present  ideas  which  provide  the 
missing  frame  of  reference  for  the  total  system  description, 
analysis  and  evaluation.  This  is  accomplished  by  quantifying 
the  component  parts  of  a  system  as  defined  in  the  first  para¬ 
graph  of  the  introduction  and  the  degree  of  relationship  between 
the  test  user,  operator,  pilot,  driver  ratings,  and  the 
measured  test  system  input  and  output  performance  data. 

This  method  allows  the  system  evaluator  to  first  opera¬ 
tionally  define  his  system  evaluation  goals  and  then  conduct  a 
systems  evaluation  which  determines  how  good  are  the  measured 
test  data  as  well  as  how  good  are  the  test  user,  operator,  pilot 
driver  ratings.  Of  utmost  importance  is  the  fact  that  the 
method  allows  the  evaluator  to  determine  by  quantifying  it 
exactly  how  good  is  the  final  system  evaluation. 
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PROCEDURE 


The  criteria  presented  for  determining  the  effectiveness 
of  a  completed  piloted  aircraft  system  evaluation  is  the  degree 
of  correlation  of  flight  test  pilot  ratings  PRP  with  engineering 
calculations  procedure  ECP  measures  .  ECP .  The  higher  the 

product-moment  correlation  ^PEP  •  E£P ,  the  better  the  engineer¬ 
ing  calculating  procedures  ECP,  selected  are  in  predicting  the 
combined  or  selected  system  characteristics  being  evaluated.  In 
like  manner,  the  higher  the  correlation  the  more  confidence 
that  may  be  placed  in  the  validity  and  reliability  of  the  flight 
test  pilot  (operator)  ratings. 

The  new  flight  test  pilot  rating  procedure,  PRP,  presented 
is  based  on  the  average  of  the  logarithms  of  the  pilot  (operator) 
magnitude  estimation  ratings  of  total  mental  and  physical  workload 
due  to  the  handling  qualities  and  performance  characteristics  of 
the  aircraft  (vehicle)  during  a  time  segment  of  a  selected  or 
required  critical  flight  (road,  sea,  space)  operation.  This 
rating,  PRP test  the  test  configuration  is  reference  a  given 

standard  aircraft  configuration  with  known  measured  handling 
qualities  and  performance  characteristics,  i.e.,  measured  and 
critical  flight  operation  which  is  given  a,  pRpstd  mental 

and  physical  workload  rating  of  10.  If  the  test  aircraft  con¬ 
figuration  requires  twice  the  amount  of  the  pilots  mental  and 
physical  workload,  in  comparison  to  the  standard  configuration, 
the  pilot  would  rate  it  as  having  a  PRP  of  20.  If  one-half,  the 
pilot  would  rate  it  as  having  a  PRP test  of  5>  etc.  If  less 
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than  one-half,  ratings  of  1  through  4  are  available  for  des¬ 
cribing  the  test  configuration  workload.  In  similar  fashion 
ratings  of  21  through  40  describe  workload  increasing  from 
double  to  four  times  the  selected  standard  workload. 

The  new  pilot  rating  procedure  is  therefore  stated 
mathematically  as  follows: 

PRP  =  10  log  (PRPtest/PRPstd^  •  decivals,  dV.  (3) 

This  method  provides  a  pilot  rating  continuum  from  zero  or  a 
rating  of  the  test  configuration  as  being  identical  with  the 
standard  configuration  to  an  infinite  number  of  ratings. 

Ratings  in  a  positive  direction  indicate  a  doubling  of  the 
pilot-aircraft  system  workload  for  each  three  decival,  dV  in¬ 
crease.  The  negative  direction  indicates  a  halving  of  the 
pilot-aircraft  system  workload  for  each  three  decival,  dV 
decrease . 

The  new  engineering  calculation  procedures  presented  may 
be  stated  mathematically  as  follows: 

ECP  =  10  log  (ECPtest/ECPst(J)  decivals ,  dV.  (4) 

The  logarithmic  units,  10  log  (ECPtest/ECPstd)  and  10  log 
(pRFtest/pRpstd)  used  in  this  system  evaluation  method  are 
termed  "decivals,  dV"  as  they  are  10  times  the  log  base  10  of 
the  ratio  of  the  selected  ECP  and  PRP  system  evaluation  units 
for  the  test  aircraft  configuration  compared  to  the  chosen 
standard  aircraft  configuration. 


In  this  way  a  doubling  or  halving  of  the  ECP  or  PRP 
of  the  test  configuration  ratio  with  the  chosen  standard 
configuration  results  in  a  ±3  decival,  3  dV,  increase  or  de¬ 
crease  as  10  times  log  2  =  10  x  0.3  =  3  dV  and  10  log  (1/2)  = 

10  x  -0.3  =  -3  dV. 

Each  ECP  is  -  elected  to  quantify  a  known  outstanding  cri¬ 
tical  characteristic  of  the  operating  system  such  as  integra¬ 
tion  of  the  pound-seconds  of  force  exerted  by  the  pilot  on  the 
flight  controls  about  the  pitch,  roll,  and  yaw  axis  of  the 
aircraft  during  the  time  segment  of  a  selected  flight  opera¬ 
tion  which  may  later  be  shown  to  correlate  highly  with  the 
pilots  ratings  of  physical  workload.  In  like  manner,  the  inte¬ 
gration  of  the  number  of  bits  of  information  transmitted  per 
second  by  the  primary  flight  instruments  during  the  time  segment 
of  the  selected  flight  operation  may  later  be  shown  to  correlate 
with  the  pilots  rating  of  mental  workload. 

Each  ECP  is  therefore  a  non-dimensional  logarithmic  ratio 
of  the  selected  critical  system  characteristic  ECP^.es^  to  the 
same  critical  system  characteristic  of  the  selected  standard 
aircraft  configuration  ECPst<j.  This  is  a  very  important  part 
of  this  new  system  evaluation  procedure  as  it  provides  a  means 
of  converting  all  critical  systems  characteristics  to  non- 
dimensional  logarithmic  ratios  of  selected  critical  test  con¬ 
figuration  characteristics  compared  to  the  same  critical 
standard  configuration  characteristics  under  identical  test 
conditions.  Note  well  that  this  non-dimensional  nature  of 
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all  the  system  ECPs  permits  the  combination  of  all  selected 
critical  system  ECPs  into  a  single  integrated  ECP  for  the 
time  segment  of  the  selected  flight  operation.  This  is 
accomplished  by  summing  them  logarithmically: 

ECP  =  10  log  (antilog  ECP^/10  +  antilog  ECP2/IO  ....+  antilog  ECPn/10)  (5) 

It  is  important  to  note  that  pilot  ratings  PRP  are  also 
non-dimensional  as  they  are  10  times  the  log  of  the  ratio  of 
the  test  aircraft  configuration  to  the  selected  standard  con¬ 
figuration,  PRptest/pRPstd*  Pilot  ratings  obtained  during  the 
same  time  segment  of  the  selected  flight  operation  for  both 
the  test  and  the  standard  aircraft  configurations,  therefore, 
provide  a  summary  PRP  test  pilot  rating  which  corresponds  to 
the  summary  ECP  flight  test  data  calculations.  It  is  the 
degree  of  correlation.^  PRp  .  g^P  these  values  obtained 
from  a  suitable  number  of  pilots  which  determines  how  effective 
the  selected  system  evaluation  ECP  measures  are  in  describing 
the  characteristics  of  the  operating  system  as  rated  by  the 
test  pilots. 

The  test  pilot  ratings  obtained  during  the  chosen  time 
period  of  the  selected  flight  operation  are  the  accepted 
criteria  for  determining  the  actual  real-life,  real-time 
characteristics  of  the  test  aircraft  in  its  flight  environment. 

As  the  PRP  and  ECP  measures  are  a  log  X  log  Y  relation¬ 
ship,  it  can  be  assumed  and  proven  subsequently  by  inspection 
that  their  relationship  is  rectilinear  and  that  a  straight 
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regression  line  equation  established  by  the  method  of  least 
squares  through  the  plotted  values  obtained  in  each  test 
flight  (or  flight  similator  test)  provides  the  answer  to  how 
good  is  the  aircraft  given  an  ECP : 


10  log 


(pBPtest/pra’std) 


k  +  n  10  log  f  BCP 


0 


■) 


(6: 


The  number  and  type  of  ECP  measures  chosen  for  an  evalu¬ 
ation  may  vary  from  one  measureable  characteristic  of  the 
piloted-aircraf t  system  made  during  the  time  period  of  the 
selected  flight  operation  to  several  which  may  be  examined 
individually,  in  pairs  and  all  possible  combinations.  The 
effectiveness  of  any  individual  ECP  measureable  piloted-aircraf t 
system  characteristic  may  be  determined  by  its  degree  of 
correlation  with  the  PRP  flight  test  pilot  ratings  obtained. 
Those  with  low  correlations  may  be  rejected  while  those  which 
correlate  highly  may  be  retained  for  further  system  evaluation. 

The  goal  is  to  obtain  through  flight  and  simulator  test 
experience  those  ECP  measures  which  provide  the  highest  corre¬ 
lation  with  the  flight  test  pilot  ratings. 

There  are  two  significant  types  of  ECP  measures  available 
for  use  by  the  systems  engineer.  One  is  a  peak  measure  which 
is  the  maximum  value  obtained  by  integration  over 
the  complete  time  period  of  the  values  sampled  approximately 
every  0.5  or  1.0  second. 
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Previous  psychoacoustic  test  experience  has  shown  that 
the  time  integrated  measures  of  cockpit  sound  pressure  and 
frequency  provide  higher  correlation  with  pilot  ratings  of 
cockpit  perceived  sound  level  than  maximum  peak  measures.  ^  The 
pilot  may,  therefore,  be  asked  to  rate  and  the  engineer 
measure  the  total  changing  situation  over  a  chosen  time  period 
of  a  selected  critical  flight  operation  as  well  as  the  peak 
value  occuring  during  that  time  period. 

Some  of  the  possible  measureable  ECP  characteristics  of 
the  piloted-aircraf t  system  made  during  the  chosen  time  period 
of  the  selected  flight  operation  include  pilot-aircraft  flight 
control  induced  physical  workload;  pilot-aircraft  primary 
flight  instrument  display  induced  mental  workload;  total  in 
flight  system  pilot-aircraft  induced  mental  and  physical 
workload;  pilot-ATC  system-aircraft  navigation  communication 
display  and  control  induced  mental  and  physical  workload; 
pilot-aircraft  flight  operational  environment  induced  workload 
(turbulence,  wind  shear,  visibility,  IFR,  VFR,  Noise,  Vibration, 
Cockpit  lighting)  and  pilot-aircraft  induced  workload  due  to 
the  aircraft  handling  quality-performance  descriptors  such 
as:  dutch-roll,  adverse  yaw,  longitudinal  stability,  lift 
over  drag  ratios  L/D  in  different  flap,  gear,  slot,  trim 
configurations,  etc.,  thrust  to  weight  ratios  T/W,  etc.  As 


they  ultimately  become  available  with  the  operation  of  all 
systems,  the  aircraft  accident,  flight  safety  statistics  over 
time  for  the  various  critical  flight  operations  such  as  takeoff 
and  approach  to  landing,  etc.,  will  provide  the  most  signifi¬ 
cant  potential  ECP. 

The  degree  of  correlation  of  any  of  the  above  ECP  measures 
with  the  PRP  pilot  ratings  obtained  during  flight  tests  or 
flight  simulator  tests  for  a  time  period  of  a  selected  flight 
operation  determines  their  suitability  for  retention  and  con¬ 
sideration  in  the  final  ECP  measures  included  in  the  final 
system  evaluation. 

All  PRP  pilot  ratings  and  ECP  calculation  procedures  are 
logarithmic  ratios  between  the  test  aircraft  configuration  and 
the  chosen  standard  of  reference  for  all  ratings  and  measures, 
a  given  standard  aircraft  configuration,  during  a  time  period 
of  a  selected  flight  operation  flown  under  identical  or  as 
constant  as  practicable  test  conditions. 

All  flight  test  pilots  will  have  experience  and  sufficient 
training  in  the  selected  standard  aircraft  configuration  during 
the  time  period  of  the  selected  flight  operation  and  constant 
flight  test  conditions  to  assure  high  reliability  and  confidence 
in  their  ratings. 

All  system  evaluation  ECP  and  PRP  measures  are  calculated  in 
logarithmic  form  and  so  may  be  integrated  over  time  for  levels 


13 


sampled  at  discrete  time  intervals  of  for  example  (0.5! 
one-half  second.  The  working  expression  becomes: 


SPRP  -  10  log 


£ECP  =  10  log 


k  =  _d_ 

At 

5. 

k  =  0 

k  =  _d_ 

At 


10 


PRP  /10 
(k) 


10  log  jAt 


(7) 


10 


ECP  /io 
(k) 


k  =  0 


10  log  /^t 


(8) 


where  d  is  the  time  duration  in  seconds  during  which  ECP  (PRP)  is 
measured  and  A t  is  the  time  interval  between  the  ECP  samples. 
Note  well  that  ECP  is  as  defined  by  Equation  Number  4,  i.e., 
10  log  (ECPtest/ECPstd). 

The  psychological  set  of  the  flight  test  pilots,  that  is 
the  operational  definition  of  the  system  evaluation  goals 
determines  the  outcome  of  the  evaluation.  The  psychological 
set  of  the  pilots  is  based  on  the  operational  definitions  used 
in  the  system  evaluation  which  define  precisely  what  is  being 
evaluated.  The  set  adopted  should  be  maintained  throughout 
for  rating  both  the  standard  and  the  test  aircraft  configura¬ 
tions.  The  possible  sets  include  but  are  not  limited  to  the 
following: 

(1)  The  whole  or  total  operationally  defined  pilot-aircraft 
system  workload  induced  by  the  aircraft  configuration 
during  a  chosen  time  period  of  a  selected  flight  opera¬ 
tion  such  as  an  XLS  or  MLS  approach  to  Dullec  Inter¬ 
national  Airport. 
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(2) 


The  operationally  defined  physical  workload  only  asso¬ 
ciated  with  movement  of  the  flight  controls  and  inte¬ 
gration  over  time  of  the  pound-seconds  of  force  about 
the  pitch,  roll  and  yaw  axis. 

'3)  The  operationally  defined  mental  workload  only  associated 
with  the  primary  flight  instruments  and  integration 
over  time  of  the  bits  of  information  transmitted  per 
second  by  these  instruments. 

(4)  The  operationally  defined  mental  and  physical  workload 
of  (3)  and  (4)  above;  combined  in  decival,  dV  system 
evaluation  logarithmic  units. 

(5)  Other  psychological  sets  operationally  defined  according 
to  the  system  evaluation  goals.  For  example:  System 
evaluation  of  any  subsystem  change  such  as:  new  active 
flight  controls;  new  primary  flight  instruments  such 

as  electronic  altitude  director  indicator  changes;  ATC 
Communications-Navigation  changes;  new  collision  avoid¬ 
ance  or  data  link  changes,  etc.,  which  change  pilot 
mental  and  physical  maximum  peak  workload  or  workload 
over  time. 

The  degree  of  effectiveness  of  the  final  system  evaluation 
resulting  from  the  selection  of  appropriate  system  engineering 
calculation  procedure  ECP  measures  in  evaluating  the  piloted 
aircraft  or  any  human  operated  system's  performance  can  be  de- 
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termined  by  the  ECP  correlation  with  the  pilot  ratings,  i.e. 
by  the  value  of  the  positive  or  negative  product  moment 


correlation  coefficient 


Xt, 


PRP 


ECP 


as  follows: 


TABLE  1.  INTERPRETATION  OF  SYSTEM  EVALUATION  RESULTS 


PRP  .  ECP 

± 

0.9  to  1.0 


0.78  to  0.89 
0.64  to  0.77 
0.46  to  0.63 
0.00  to  0.45 


SYSTEM  EVALUATION  RESULTS 
Very  High 
High 

Moderate 

Low 

Very  Low 


PRP  ■  ECP* 
81  to  100% 

61  to  80% 

41  to  60% 

21  to  40% 

0  to  20% 


*The  amount  of  variance  in  test  pilot  ratings  PRP  that 
is  common  to  or  that  is  in  agreement  with  the  variance 
in  the  chosen  engineering  calculation  procedure  ECP 
test  data. 


A  detailed  explanation  of  the  product-moment  correlation 
coefficient  and  the  interpretation  of  correlation  upon  which 
the  above  Table  1  is  based  is  contained  in  Reference  6, 

Pages  79-106. 

An  example  of  system  evaluation  results  and  the  conclusions 
that  may  be  drawn  are  presented  in  Figure  1. 

An  example  of  the  major  steps  to  be  taken  in  conducting 
a  systems  evaluation  is  contained  in  Appendix  B. 


SYSTEM  EVALUATION:  Pilot  Workload 
and  Aircraft  Handline  Qualities 


. . .  +  A-M 


-+- 


~t  t 


mm 

.~U  f  f  + 

U  i  i  .  i 

W-  ‘  •  • 


Figure  1  is  an  example  of  system  evaluation  results  and 
the  following  is  an  example  of  conclusions  which  may  be 
drawn t 

The  expert  flight  test  pilot  ratings,  PRP  correlated 
highly,  ___  ___  ”  0.90  with  the  measured  flight  data 

ECP.  PRP  *  ECP 

This  result  shows  that  over  80%  of  the  variance  in  pilot 
ratings  PRP  is  associated  with  the  variance  in  the  flight 
test  data  ECP  which  indicates  a  very  high  qualitative 
system  evaluation.  The  system  evaluation  places  the  test 
configuration  as  being  twice  as  good  or. requiring  only 
one-half  the  pilot-aircraft  system  workload  as  the  com¬ 
parison  standard  configuration  during  instrument  landing 
approaches  under  identical  flight  conditions. 
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DISCUSSION 


The  chosen  or  'given'  standard  aircraft  configuration  is 
selected  for  comparison  purposes  of  both  the  pilot  or  operators 
ratings  and  the  flight  test  measures  obtained  for  the  test 
configuration.  If  the  comparison  or  standard  aircraft  con¬ 
figuration  is  one  with  which  the  test  pilots  are  already 
familiar  and  readily  available  for  flights,  this  simplifies 
both  the  choice  of  a  standard  and  the  problem  of  learning  its 
performance  characteristics. 

The  standard  configuration  is  not  presented  as  anything 
other  than  a  baseline  for  comparison  with  the  test  aircraft 
configuration  for  both  pilot  rating  PRP  and  engineering  calcu¬ 
lation  procedure  ECP  purposes.  It  provides  the  potential  means 
by  which  various  flight  test  data  as  different  as  apples  and 
oranges  are  turned  into  non-dimensional  logarithmic  ratio  units 
termed  decivals  dv  and  thereby  these  at  first  seemingly  dif¬ 
ferent  types  of  measures  may  be  combined  by  logarithmic  summa¬ 
tion. 

This  provides  the  opportunity  to  ultimately  discover  by 
repeated  tests  and  exercise  of  this  system  evaluation  method 
the  most  important  ECP  measures.  These  measures  will  be  the 
sum  of  the  flight  test  measured  data  parts  which  make  up  the 
whole  pilot  rating  of  the  aircraft  performance.  These  most 
important  flight  test  measures  when  combined  have  the  highest 
correlation  with  the  test  pilot  ratings. 
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Pilot  ratings  obtained  during  flight  tests  may  be  ratings 
of  maximum  peak  workload  which  occurs  during  the  selected  flight 
segment  such  as  an  instrument  landing  system  ILS  approach.  It 
may  also  be  necessary  to  obtain  the  pilot  ratings  of  workload 
obtained  at  different  times  on  different  flight  segments.  In 
this  case  a  pilot  rating  of  total  workload  integrated  over  time 
may  be  computed  by  using  Equation  7. 

The  correlation  of  the  PRP  ratings  with  Cooper-Harper 
pilot  ratings  may  also  be  determined.  It  will  be  possible  to 
then  compare  the  Cooper-Harper  ratings  obtained  on  the  comparison 
or  standard  aircraft  with  those  for  the  test  aircraft.  Using 
the  logarithm  of  the  Cooper-Harper  ratings  will  permit  the 
checking  of  the  flight  test  results  and  the  possibility  that 
the  present  1  through  10  rating  scale  is  somewhat  logarithmic. 

A  regression  line  established  by  the  method  of  least  squares 
will  quantify  the  log  linear  relationship  between  the  two  scales. 
Preliminary  paper  and  pencil  tests  (7)  show  that  their  is  a 
logarithmic  linear  relationship  between  the  pilots  or  rater's 
judgement  of  workload  doubling  and  the  Cooper-Harper  scale 
especially  in  the  mid-scale  4,  5,  and  6  ratings.  There  is  pre¬ 
sently  not  enough  workload  described  by  ratings  near  the 
good  end  (present  ratings  of  1,  2,  and  3)  and  possibly  too  much 
at  the  bad  end  (7,  8,  9,  and  10)  to  meet  a  perfect  logarithmic 
linear  relationship  with  the  raters  judgement  of  the  rate  at 
which  workload  is  doubling  as  presently  described  by  the  1 
through  10  descriptive  language  associated  with  each  numerical 
rating.  (Test  (7)  results  are  contained  in  Appendix  A.) 
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This  could  be  achieved  by  psychometric  scaling  of  the 
descriptive  language  and  should  be  considered  in  future  modi¬ 
fications  of  these  types  of  pilot  ratings  which  use  descrip¬ 
tive  language.  The  descriptive  language  in  each  of  the 
rating  choices  1  through  10  would  be  0.1  one-tenth  logarithmic 
distance,  i.e.  relative  rate  of  increasing  workload  as  follows: 


.1 

.2 

.3  .4 

.5 

.6 

.7 

.8 

.9 

1.0 

1.26 

1.58 

2.  2.5 

3.16 

4.0 

5.0 

6.31 

8.00 

10.0 

(Relative 
Rate  of 
Increasing 
Workload) 

Note  that  in  this  scale  the  workload  doubles  every  third 
descriptive  rating  of  workload  and  aircraft  handling  qualities. 

It  is  interesting  to  note  that  when  workload  has  increased 
10  times  over  some  minimum  or  ideal  workload  situation  that  it 
is  a  disaster  and  the  aircraft  is  uncontrollable.  Workload 
must  be  kept  at  levels  ideally  below  the  doubling  and  certainly 
the  quadrupling  of  the  minimum  or  ideal  workload  level  where 
R  =  1.0  and  log  R  =  0 . 

This  emphasizes  the  need  to  increase  the  number  of  descrip¬ 
tive  ratings  presently  available  to  the  test  pilots  at  the 
good  or  low  workload  end  of  the  scale  and  the  development  of  a 
truly  logarithmic  descriptive  language  scaling  of  workload 
and  handling  qualities  of  aircraft. 
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The  ideas  regarding  system  evaluation  and  pilot  ratings 
presented  here  are  intended  to  supplement  rather  than  replace 
the  present  Cooper-Harper  system  of  rating  aircraft  and  both 
types  of  pilot  ratings  would  be  obtained  as  well  as  other 
methods  not  discussed  herein  which  then  may  be  compared. 


In  fact  to  test  this  method,  the  following  may  be  carried 
out  not  only  for  future  tests  but  in  the  analysis  of  past 
recorded  flight  test  results  which  include  both  pilot  ratings 
and  measures  which  are  available  for  two  or  more  aircraft. 


The  decival  values  would  be  computed  for  the  pilot  ratings 
and  flight  test  measures  using  one  aircraft  chosen  arbitrarily 


as  the  comparison  or  standard  and  the  other  as  the  test  air¬ 


craft.  The  decival  value  of  10  log 


'test 

Standard 


is  determined 


for  each  test  pilot  and  compared  with  the  decival  value  of 


10  log 


^test 

ECPstandard 


for  appropriate  measured  flight  test  data. 


The  degree  of  correlation  of  the  decival  pilot  ratings  with 


the  decival  flight  test  measures  will  aid  in  the  choice  of 
future  flight  test  measurements  and  procedures. 


Where  changing  environmental  variables  are  of  concern 
as  to  their  effects  on  the  system  including  pilot  or  operator 
workload  and  the  handling  qualities  of  the  aircraft  or  vehicle 
then  the  test  variable  becomes  the  logarithmic  ratio  of  an 
environmental  change  with  time  compared  to  a  chosen  standard 
condition . 


The  decival  value  of  the  environmental  change  ENV  = 


10  log 


'test  |  may  be  computed  and  compared  with  the  PRP 


decival  pilot  ratings  by  computing  their  correlation  coefficient, 
PRP  •  ENV,  obtained  for  these  same  environmental  test  and 
standard  conditions.  The  environmental  conditions  may  be  man 
made  or  natural  for  example:  changing  airport,  runway,  approach 
lighting  methods,  or  the  amount  of  turbulence  or  wind-shear 
for  repeated  ILS  approaches  by  the  same  piloted-aircraf t .  For 
road  vehicles,  one  may  test  the  differences  in  roadways  and 
strive  to  learn  from  driver  ratings  and  handling  quality  mea¬ 
sures  why  there  are  more  accidents  for  left-hand  highway  curves 
compared  to  right-hand  curves.  Other  examples  of  conditions 
which  might  be  tested  in  this  way  include  the  effects  of  dif¬ 
ferent  kind  of  roadway  signs  and  signal  methods  as  well  as  the 
effects  of  weather  such  as  rain,  snow,  ice  on  the  road  safety 
of  the  national  highway  system. 
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Regarding  future  system  models,  it  is  interesting  to  note 
that  the  relationship  between  pilot  response  and  system  output 
measures  having  high  correlation  with  each  other  based  on  the 
proposed  system  evaluation  method  may  be  used  in  developing 
and  improving  system  models. 

In  the  crossover,  optimum  control  and  other  system  models 

decivals  and  the 

I ECP  1 

system  output  is  expressed  as  10  log  I  test!  decivals. 

lfcpstd  J 

The  nature  of  the  units  of  pilot  response  and  system  output 
flowing  through  the  system  model  are  non-dimensional  logarithmic 
ratios  of  pilot  response  and  system  output  of  the  system  being 
modeled,  i.e.,  the  test  system  compared  to  a  "given"  or  chosen 
standard  system.  This  represents  how  the  pilot  or  operator 
responds  to  the  displays  of  system  output  in  the  feedback 
loop.  He  is  comparing  how  the  new  or  test  aircraft,  vehicle, 
machine  or  tool  responds  compared  to  a  known  or  learned  standard 
system  with  which  he  has  prior  experience.  This  is  suggestive 
of  an  approach  to  also  modeling  how  the  operator  learns  based 
on  his  experience  with  one  system  compared  to  another. 


pilot  response  is  equal  to  10  log 


test 


The  degree  of  proficiency  of  the  operators  of  systems 
being  evaluated  is  usually  a  problem.  In  the  present  method 
the  test  subjects  may  have  varying  degrees  of  proficiency 
as  they  will  be  rating  their  workload  on  a  test  system  relative 
to  a  standard  system.  Their  ratings  are,  therefore,  relative 
to  the  same  degree  of  proficiency  used  to  perform  the  selected 
operation  evaluated  for  both  the  standard  system  and  the  test 
system.  All  test  operator  ratings  and  test  measures  are  thexefo 
equally  useful  for  system  evaluation  purposes  as  long  as  the 
degree  of  proficiency  is  such  that  the  test  operators  can 
perform  the  selected  operations  being  evaluated  from  begin¬ 
ning  to  end  on  the  test  and  standard  systems. 

Another  advantage  of  this  system  evaluation  method  is  that 
it  can  be  used  to  assure  that  new  evolving  systems  are  im¬ 
provements  over  previous  systems  as  to  their  handling  qualities 
and  operator  workload.  These  are  factors  which  have  a  direct 
bearing  on  safety.  The  improvement  or  departure  of  new 
systems  from  old  systems  with  known  records  of  safety  can 
easily  be  established  early  in  the  design  stage.  Continued 
use  of  this  method  will  identify  those  engineering  calculation 
procedures  or  measures  which  have  correlated  the  best  with 
past  system  performance  ratings.  These  will  be  the  best 
predictors  of  the  future  systems  performance.  Later  in  the 
design  process  the  method  may  be  used  in  simulation  tests  and 
continued  with  tests  of  the  prototype  and  first  operational 
vehicles. 
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AN  EXAMPLE  OF  A  SYSTEM  EVALUATION 


The  following  example  is  one  made-up  by  the  author  to  deronstrate  the 
system  evaluation  procedure  more  clearly  than  mere  words.  Each  step 
is  outlined  and  pilot  ratings  and  measured  data  were  selected  to 
demonstrate  an  evaluation  in  which  there  is  a  valid  relationship  be¬ 
tween  the  pilot  ratings  and  the  test  data,  pound-seconds  of  force. 

This  is  an  example  only  and  not  to  be  interpreted  as  a  valid  real- 
world  relationship. 

It  is  interesting  to  note  that  the  first  run  of  pilot  ratings  versus 
the  test  data  yielded  a  -0.39  correlation.  This  demonstrated  that  the 
first  selection  of  paired  ratings  and  test  data  was  indeed  a  random 
sample  and  there  was  no  significant  correlation  coefficient  or  rela¬ 
tionship  between  them.  This  may  be  the  outcome  of  a  real-world 
test  of  this  method.  But  it  is  equally  as  important  to  determine 
that  there  is  not  a  relationship  between  pilot  ratings  of  what  is 
being  evaluated  and  the  selected  test  data.  It  eliminates  that 
measured  system  variable  and  the  investigator  must  search  for  those 
measured  system  variables  which  do  correlate  with  the  pilots  ratings- 
As  these  significant  relationships  are  determined  they  may  be  used  by 
everyone  and  system  evaluation  will  become  increasingly  simpler  and  more 
realistic  than  it  is  today. 
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AN  EXAMPLE  OF  A  SYSTEM  ENGINEERING  EVALUATION 


The  systems  engineering  evaluation  is  of  the  physical  workload/handling 
qualities  of  a  simulator  versus  the  physical  workload/handling  qualities 
of  the  aircraft  simulated. 

Background  --  Because  of  undesirable  pilot  induced  oscillations 
found  in  the  first  simulator  model  the  control  forces  required  in  the 
simulator  during  an  approach  were  increased  by  the  designer  over  those 
found  in  the  aircraft.  This  evaluation  is  to  determine  first  whether 
this  difference  in  the  simulator  workload/handling  qualities  and  the 
simulated  aircraft  is  a  significant  variable  in  pilot  transition 
training  which  is  detected  by  the  pilots.  Of  equal  importance  is  to 
determine  if  pound-seconds  of  force  is  a  useful  measure  which  correlates 
highly  with  the  pilots  ratings  and  which  therefore,  may  be  used  as  a 
prediction  in  future  aircraft  design  and  evaluation  by  the  Federal 
Aviation  Administration  (FAA)  and  the  Industry. 

Procedure  — 

Step  1 :  The  first  step  is  to  define  operationally  what  is  being 
evaluated  which  sets  the  system  engineering  evaluation  goals. 

System  engineering  evaluation  goal:  Do  the  control  forces  in  the 
simulator  affect  pilot  transition  training  to  the  simulated 
aircraft?  Specifically  are  the  increased  control  forces,  measured 
in  pound-seconds  integrated  over  time,  required  in  the  simulator 
to  avoid  pilot  induced  oscillations  during  a  standard  instrument 
approach  noticed  and  rated  as  such  by  the  pilots  after  they  have 
successfully  transitioned  to  the  aircraft  and  obtained  actual 
flight  experience?  Operationally  defined  the  question  is  as 

follows:  What  are  the  pilot  ratings  of  physical  work- 
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load  in  the  simulator  during  a  standard  instrument  cpproach  compared 
to  a  given  physical  workload  rating  of  10  in  the  aircraft?  And  most 
important: Is  there  a  correlation  between  the  chosen  measure,  total 
pound-seconds  of  force  integrated  over  timp.and  the  pilot  ratings 
of  physical  workload? 

Step  2:  Select  the  aircraft  standard  configuration- -This  :s  the 

real  aircraft  configuration  in  flight  during  a  standard  instrument 
approach.  The  standard  is  selected  for  comparison  purposes  only. 

It  is  the  configuration  which  has  been  incorporated  in  the  simu¬ 
lator  and  it  is  the  one  which  the  pilots  are  most  familiar  with 
and  obtained  flight  experienced  in  and  is  readily  available  for 
flight . 

Step  3:  Select  the  flight  test  engineering  calculation  procedure 
ECP,  measures  that  will  be  recorded:  The  pound-seconds  of  force 
required  about  the  pitch,  roll  and  yaw  axes  will  be  integrated  over 
time  to  obtain  objective  measures  of  the  forces  actually  required: 

(1)  To  fly  the  selected  aircraft  standard  configuration  in  actual 
flight  and  (2)  in  the  simulator  from  the  outer  marker  to  touch¬ 
down  and  roll-out  during  a  standard  instrument  approach  to  the 
local  airport. 

Step  4:  Train  the  pilots  so  that  they  understand  what  is  being 
rated  as  defined  in  Step  1:  What  are  the  magnitude  estimation 
pilot  ratings  of  physical  workload  in  the  simulator  during  a 
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standard  instrument  approach  compared  to  a  given  physical  workload 
rating  of  ten  in  the  aircraft  standard  configuration  selected  in 
Step  2  and  flown  under  similar  conditions  on  a  standard  instrument 
approach?  Pilots  are  instructed  to  rate  the  simulator  physical 
workload  as  five  if  one-half  that  of  the  aircraft  physical  workload 
and  20  if  doubled  with  any  rating  from  zero  to  infinity  being  avail¬ 
able  to  quantify  their  rating  of  simulator  physical  workload  in 
relation  to  the  selected  standard  aircraft  configuration  physical 
workload  given  a  rating  of  ten. 

Step  5:  Ten  pilots  fly  the  selected  standard  aircraft  configuration 

and  learn  during  three  instrument  approaches  that  the  operationally 
defined  physical  workload  required  to  fly  the  standard  instrument 
approach  from  the  outer  marker  to  touchdown  and  roll  out  is  given 
a  magnitude  estimation  value  of  ten. 

Step  6:  The  same  ten  pilots  fly  three  identical  instrument  approaches 
in  the  simulator  which  is  being  evaluated.  Each  pilot  provides  his 
magnitude  estimation  judgement  of  the  physical  workload  required 
to  fly  the  flight  simulator  with  its  known  simulated  aircraft 
characteristics  from  the  outer  markeT  to  touchdown  and  rollout 
compared  to  the  physical  workload  required  to  fly  the  actual 
aircraft  under  identical  conditions  given  a  physical  workload  rating 
of  ten.  If  in  the  pilot's  judgement  the  physical  workload  in  the 
simulator  is  double  that  required  by  him  to  fly  the  actual  approaches 
his  rating  is  20.  If  one-half  his  rating  is  five.  Note  well:  The 
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pilot  nay  use  any  rating  from  rero  to  infinity  to  describe  and 
most  important  quantify  their  ratings  on  a  continuous  scale 
which  is  anchored  to  a  known  and  quantified  standard  physical 
workload . 

Step  7:  Record  the  selected  test  data  during  the  actual  flights 
and  the  simulated  approaches  so  that  they  may  be  quantified  (ECP) 
and  compared  to  the  pilot  ratings  (PRP)  and  their  exact  relation¬ 
ship  to  each  other  determined,  i.e.,  ^PRP  .  ECP  will  equal  some 
number  between  -  1.0  and  1.0  which  quantifies  the  degree  of 
relationship  to  each  other  determined  between  the  selected  and 
recorded  test  data  (ECP)  and  the  pilot  judgements  (PRP) . 

The  selected  test  data  recorded  on  all  approaches  in  the  aircraft 
and  simulator  are  the  total  pound-seconds  of  force,  measured  by 
electrical  transducers,  about  the  pitch,  roll  and  yaw  axes  inte¬ 
grated  over  time  resulting  from  the  pilots  control  movements 
from  the  outer  marker  to  touchdown  and  rollout. 

Step  8:  Compute  for  each  flight  and  each  simulated  approach  the 
total  pound- seconds  of  force  measured  about  the  three  axes  from 
the  outer  marker  to  touchdown  and  rollout.  Determine  (1)  the 
total  pound-seconds  of  force  for  each  pilot  during  the  three 
actual  flights  and  (2)  the  total  for  each  pilot  for  the  three 
flights  in  the  simulator,  (3)  the  average  total  pound-seconds  of 
force  for  each  pilot  and  the  ratios  of  the  force  required  in  the 
simulator  compared  to  that  required  in  the  aircraft. 


5 


Force  (pound-seconds) 


Siuulator 


Force  (pound -seconds) 

Aircraft 

(4)  Calculate  the  logarithm  of  this  ratio  and  multiply  by 
ten  to  obtain  the  ECP  value. 


ECP  »  10  log 


10 


Force  (pound- seconds) 


Simulator 


Force  (pound-seconds) 


Aircraft 


The  ECP  is  the  Engineering  Calculation  Procedure  used  to  obtain  a 
logarithmic  continuous  non-dimensional  measure.  (See  example,  Table  1) 


Step  9:  In  a  similar  fashion  to  Step  8  (1)  Record  foT  each 

simulator  test  flight,  the  pilot's  magnitude  estimation  of  the 
physical  workload  required  to  fly  the  simulator  (2)  determine 
for  each  pilot  the  ratio  of  these  magnitude  estimation  ratings 
compared  to  the  aircraft's  given  magnitude  estimation  value  of 
10. 


Magnitude  Estimation 
(0  to  00) _ 


Rating 

Simulator 


Given  Magnitude 
(10) 


Estimation  Rating 
Aircraft 


(3)  Calculate  the  logarithm  of  this  ratio  and  multiply  by  10. 

This  is  the  pilot  rating,  PRP  value  entered  in  the  last 
column  of  Table  2  for  each  pilot  A  through  J. 

PRP  ■  10  log10  (*-E-  Ratin*simulator/31ven  M-E-  Rating  Aircraft) 
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AN  EXAMPLE  OF  SYSTEM  ENGINEERING  EVALUATION  TEST  DATA.  AND  RESULTS 

Engineering  Calculation  Procedures,  ECP  —  The  following  data  continued 
in  Table  1  are  the  measured  test  data  obtained  from  electrical  transducers 
which  measured  the  total  pound-seconds  of  force  required  about  the  pitch, 
roll  and  yaw  axes  recorded  during  simulator  and  aircraft  flights  from 
the  outer  marker  to  touchdown  and  roll  out. 

Table  1  shows  how  the  measured  data  are  averaged  in  both  the  simulator 
and  aircraft  and  their  ratios  obtained,  simulator:  aircraft,  and 
converted  into  10  log  form  to  obtain  ECP  values. 
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TOTAL  POUND-SECONDS  OF  FORCE 


7 


Pilot  Rating  Procedures,  PRP  —  The  following  date  contained  in  Table  2. 
are  the  pilot  magnitude  estimation  ratings  of  pilot  physical  workload 
compared  to  aircraft  physical  workload  given  a  magnitude  estimation 
rating  of  ten.  Ratings  are  obtained  from  the  same  ten  pilots  listed 
in  Table  1  as  pilots  A  through  J. 

The  ratio  of  the  average  magnitude  estimation  of  physical  workload  in 
the  simulator  for  each  pilot  compared  to  the  magnitude  estimation  of 
physical  workload  in  the  aircraft  given  a  value  of  ten  is  calculated 
for  each  pilot.  Ten  times  the  logarithm  of  this  average  ratio£ 10  log 
(average  ratio)J gives  the  pilot  rating  procedure,  PRP,  value  for  each 
pilot  A  through  J. 
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The  relationship  or  correlation  between  pilot  ratings,  PRP  and  the 
engineering  measures,  ECP,  may  now  be  obtained.  From  this  quantified 
relationship,  two  equations  based  on  the  least  squares  fit  of  the 
PRP  and  ECP  data  may  be  obtained  so  as  to  predict  one  of  these  measures 
based  on  the  other. 

The  magnitude  of  the  relationship  may  be  compared  to  a  table  of 
correlation  coefficients,  such  as  Table  D  in  Reference  6- 
According  to  the  number  of  degrees  of  freedom,  N  pairs-lsnine  in  the 
present  case,  correlations  larger  than  0.602  and  0.735  could  occur 
by  chance  and  chance  alone  only  five  and  one  time  in  a  hundred  re¬ 
spectively. 

Table  3  presents  an  example  of  this  process. 
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TABLE  3.  THE  TEST  OF  RELATIONSHIP  OF  MEASURED  TEST  DATA, 

ECP  WITH  PILOT  RATINGS,  PRP 


NUMBER  OF 

AVERAGED 

PAIRS 

PRP 

ECP 

RESULTS 

1 

2.553 

3.01 

^  PRP  .  ECP  =  0.89* 

2 

2.3 

2.553 

The  correlation  of  pilot  ratings  with  measured 
flight  test  data. 

3 

3.424 

3.29 

Eq.l.  PRP  =  0.647  ♦  1.161  (ECP) 

4 

2.553 

3.01 

Used  to  predict  pilot  ratings,  PRP,  based  on 
measured  flight  test  data,  ECP. 

5 

2.3 

2.34 

Eq.2.  ECP  =  1.07  +  0.68  (PRP) 

6 

3.424 

3.554 

Use  to  predict  measured  flight  test  data,  ECP 
based  on  pilot  ratings,  PRP. 

7 

3.22 

3.144 

yt  2  -  0.892  =  80* 

8 

3.01 

3.01 

Common  variance  in  ratings  and  flight  test  data 

9 

3.22 

3.29 

*  Statistical  tables  such  as  Table  D  of 

10 

2.3 

2.75 

Reference  6  show  that  for  10  pairs  of  data 
the  correlation  must  be  greater  than  0.735 

to  be  99  percent  confident  that  the  correlation 
is  real  and  did  not  occur  by  chance  and  chance 
alone.  Because  of  the  0.89  correlation  the 
system  evaluation  is  valid  and  Equation  1  will 
be  useful  to  predict  pilot  ratings  of  physical 
workload  based  on  measured  or  calculated 
total  pound-seconds  of  force  values. 
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CONCLUSIONS 


A  system  evaluation  method  is  presented  which  systematizes 
and  quantifies  both  PRP  pilot  rating  procedures  and  ECP 
engineering  calculation  procedure  measures  of  system  performance 
on  a  logarithmic  ratio  basis  of  test  aircraft  configurations 
compared  to  a  known  selected  standard  aircraft  (vehicle)  con¬ 
figuration.  The  logarithmic  units  10  log  (ECPtest/ECPstd)  and 
10  log  (pRPtest//pRPstd^  usec*  i-n  this  system  evaluation  method 
are  termed  "decivals,  dV"  as  they  are  10  times  the  log  base 
10  of  the  ratio  of  the  ECP  and  PRP  values  obtained  during 
tests  for  the  test  aircraft  configuration  compared  to  the 
chosen  standard  aircraft  configuration. 

The  system  evaluation  is  for  chosen  time  periods  of 
selected  flight  operations  which  are  critical  to  flight  safety, 
such  as  may  occur  during  takeoff,  or  approach  to  landing  and 
may  include  emergency  engine  failure,  flight  control  or  instru¬ 
ment  malfunction  conditions.  System  equations  are  presented 
which  answer  the  question  as  to  how  good  is  the  test  configura¬ 
tion  in  relation  to  the  known  standard  configuration  during 
these  same  flight  conditions.  Potential  ECP  measures  are 
discussed  and  their  correlation  with  PRP  pilot  ratings  obtained 
during  flight  test  or  flight  simulator  test  determines  their 
retention  as  effective  system  performance  and  evaluation 
measures.  The  non-dimensional  logarithmic  nature  of  the  re- 
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tained  ECP  system  performance  descriptors  allows  their  com¬ 
bination  by  logarithmic  summation  and  their  correlation  with 
the  PRP  pilot  ratings  is  determined.  The  combination  of  ECP 
measures  having  the  highest  correlation  with  pilot  ratings 
is  retained  for  final  system  evaluation. 
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APPENDIX 


Two  tests  were  conducted  where  32  subjects,  11  pilots, 
and  21  non-pilots  rated  the  increasing  workload  based  on  the 
Cooper-Harper  descriptive  language  regarding  aircraft 
characteristics,  demands  on  the  pilot,  and  adequacy  of  the  air¬ 
craft  for  selected  task  or  required  operation.  The  required 
operation  was  limited  to  takeoff,  landing  or  cruise  under 
visual  flight  rules  and  non-turbulent  weather. 

In  test  one,  the  32  subjects  estimated  the  increasing 
workload  for  each  Cooper-Harper  pilot  rating  one  through  10 
reference  a  standard  aircraft  with  a  pilot  rating  of  one,  i.e., 
a  C-H  rating  of  one.  This  C-H  rating  number  one  was  given  a 
standard  workload  rating  of  10.  Where  the  aircraft  character¬ 
istics  and  demands  on  the  pilot  are  doubled  the  subjects  gave 
that  C-H  number  a  rating  of  20.  When  that  workload  doubled 
they  entered  40  and  80  for  the  next  doubling,  etc. 

In  test  two,  conducted  60-days  later,  the  32  subjects 
rated  the  increasing  and  decreasing  workload  but  Cooper- 
Harper  pilot  rating  6  was  given  a  workload  standard  rating 
of  10.  The  test  subjects  rated  as  in  test  one  but  also  where 
workload  was  one-half  they  entered  5  and  one-half  of  that 
workload  they  entered  2.5,  etc. 
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APPENDIX  (Cont’d) 


The  hypothesis  tested  and  the  results  of  the  tests  were 
as  follows: 


1  2  3  4  5  6 

Hypothesis  Tested  (Log  Workload  Ratio) : 

7 

8 

9 

10 

0  0.1  0.2  0.3  0.4 

Test  1  (Log  Workload  Ratio) : 

0.5 

0.6 

0.7 

0.8 

0.9 

0  0.04  0.176  0.3  0.43 

Standard 

Test  2  (Log  Workload  Ratio) : 

0.57 

0.76 

0.92 

1.1 

1.25 

-0.04  -0.16  0.11  0.3  0.462 

Hypothesis  Tested  (Workload  Ratio) : 

0.57  0.84 

Standard 

1.0 

1.28 

1.55 

1  1.25  1.58  2  2.5 

Test  1  (Workload  Ratio) : 

3.16 

4. 

5. 

6.5 

8.0 

1  1.1  1.5  2  2.7 

Standard 

Test  2  (Workload  Ratio) : 

3.7 

5.7 

8.3 

12.7 

18.0 

0.37  0.65  1.3  2  2.9 

3.7 

Standard 

6.9 

10.4 

19.0 

35.5 

The  rating  of  the  pilots  was  not  significantly  different  from 
the  non-pilots. 
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APPENDIX  B 


Steps  In  Systems  Evaluation 

1.  Set  the  system  evaluation  goals:  Operationally  define  what 

is  being  rated.  (See  Page  14,  Section  (1)  through  Section  (5) 
for  examples.) 

2.  Select  the  aircraft  standard  configuration  for  comparison  pur¬ 
poses  only.  It  is  one  with  which  the  raters  are  most  familiar 
and  is  readily  available  for  flight.  (See  Page  18  for  dis¬ 
cussion.  ) 

3.  Train  the  raters  regarding  what  is  being  rated  and  what  the 
system  evaluation  goal  is  using  operational  definitions  and 
the  magnitude  estimation  technique  reference  a  standard 
workload  (or  what  is  being  evaluated  as  defined  in  Step  1) 

of  10  for  the  comparison  system.  (See  Page  14  for  discussion.) 

4.  Select  the  flight  test  data  ECP  measures  that  will  be  moni¬ 
tored  and  recorded  regarding  test  and  standard  system  per¬ 
formance.  (See  Page  9  for  discussion  of  ECP  examples.) 

5.  Raters  fly  the  selected  standard  aircraft  on  the  chosen  time 
segment  of  a  required  flight  operation  such  as  an  ILS 
approach  and  learn  that  the  operationally  defined  workload 
(or  what  is  being  evaluated  as  defined  in  Step  1)  is  equal 
to  10. 

6.  Raters  fly  the  test  aircraft  on  the  same  time  segment  of  the 
chosen  flight  operation  such  as  an  ILS  approach  and  obtain 
pilot  ratings  of  the  operationally  defined  workload  (or 
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what  is  being  rated)  set  in  Step  1.  Use  magnitude  estima¬ 
tion  to  judge  how  workload  compares  to  the  arbitrarily 
chosen  but  learned  standard.  If  double,  rate  it  20,  if 
one-half  rate  it  5,  etc.  (See  Page  7  for  discussion.) 

7.  Record  selected  flight  test  data,  ECP  measures,  during  the 
operation  of  both  the  test  and  standard  aircraft  for  which 
pilot  ratings,  PRP  magnitude  estimates  of  workload,  are 
obtained  and  a  standard  operationally  defined  workload  (or 
what  is  being  rated  as  set  in  Step  1)  of  10  set  for  the 
standard  aircraft. 


8.  Compute  pilot  magnitude  estimates  of  workload: 


10  log 


"test 

6std_ 


in  decivals,  dv. 


9.  Compute  system  flight  test  data  measures: 


ECP  -  10  log 


’test  I  in  decivals,  dV  for  each  selected 


important  system  variable  selected  in  Step  4. 

10.  Determine  the  quantified  relationship  ranging  from  zero  to 
plus  or  minus  1,  i.e.  the  product-moment  correlation 
coefficient^  PRP  .  ECP  between  all  the  pilot  ratings  deter¬ 
mined  in  Step  8  with  the  corresponding  flight  test  data 
measures  selected  in  Step  9.  (See  Reference  6  for  detailed 
explanation  of  the  correlation  coefficient.) 
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11. 


12. 


Retain  those  measures  which  correlate  highly  with  the  pilot 
ratings.  Discard  those  which  lack  correlation.  Combine 
by  logarithmic  summation  those  measures  which  correlated 
best  with  pilot  ratings.  (Independent  ECP  measures) 

Compute  the  correlation  between  the  combined  ECP  measures 


retained  and  pilot  ratings  where  PRP  .  ECP  equals  +1.0 
is  perfect  correlation  and  zero  is  none.  (See  Page  16 
for  discussion  of  interpretation  of  ^*PRP  .  ECP  and  system 
evaluation  results.) 

This  is  the  degree  of  effectiveness  of  the  system  evaluation 
and  provides  a  check  on  the  predictive  value  of  the  flight 
test  data  as  well  as  the  reliability  of  the  pilot  ratings, 

PRP.  As  more  system  evaluations  using  the  method  are  com¬ 
pleted  there  will  be  a  growing  identification  of  which  flight 
test  data  measures,  ECP,  are  the  best  predictors  of  the  opera¬ 
tionally  defined  system  evaluation  goals  set  in  Step  1. 
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